Add comprehensive benchmarks for AiDotNet library #473
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…tions, and Statistics
…CrossValidation, and Internal Comparisons
…prehensive coverage areas (Interpolation, Wavelets, WindowFunctions, RBF)
…d all 35 optimizers
…arning, LoRA, RAG, and GeneticAlgorithms
…benchmarks, all features covered
…bility, and OutlierRemoval
- FitDetectorsBenchmarks.cs: 20 benchmarks for overfitting/underfitting detection
* Default, CrossValidation, Adaptive, Ensemble detectors
* Residual-based: ResidualAnalysis, ResidualBootstrap, Autocorrelation
* Statistical: InformationCriteria, GaussianProcess, CookDistance, VIF
* Resampling: Bootstrap, Jackknife, TimeSeriesCrossValidation
* Feature analysis: FeatureImportance, PartialDependencePlot, ShapleyValue, LearningCurve
* Classification: ROCCurve, PrecisionRecallCurve
- FitnessCalculatorsBenchmarks.cs: 27 benchmarks for model evaluation
* Error-based: MSE, RMSE, MAE, Huber, ModifiedHuber, LogCosh, Quantile
* R-squared: RSquared, AdjustedRSquared
* Classification: CrossEntropy, BinaryCrossEntropy, CategoricalCrossEntropy,
WeightedCrossEntropy, Hinge, SquaredHinge, Focal
* Specialized: KL-Divergence, ElasticNet, Poisson, Exponential, OrdinalRegression
* Similarity: Jaccard, Dice, CosineSimilarity, Contrastive, Triplet
- InterpretabilityBenchmarks.cs: 16 benchmarks for model explainability
* Fairness evaluators: Basic, Group, Comprehensive
* Bias detectors: DemographicParity, DisparateImpact, EqualOpportunity
* Explanation structures: LIME, Anchor, Counterfactual
* Helper metrics: UniqueGroups, GroupIndices, PositiveRate, TPR, FPR, Precision
- OutlierRemovalBenchmarks.cs: 16 benchmarks for outlier detection
* Algorithms: None, ZScore, IQR, MAD, Threshold
* Both Matrix and Tensor support
* Different threshold configurations (strict/lenient)
- CachingBenchmarks.cs: 12 benchmarks for caching performance * ModelCache operations: CacheStepData, GetCachedStepData, ClearCache, GenerateCacheKey * GradientCache operations: CacheGradient, GetCachedGradient, ClearCache * DeterministicCacheKeyGenerator: GenerateKey with/without parameters, CreateInputDataDescriptor * Cache hit/miss patterns for both ModelCache and GradientCache * Tests concurrent access patterns and key generation performance - SerializationBenchmarks.cs: 17 benchmarks for JSON serialization * Matrix serialization: Serialize, Deserialize, RoundTrip * Vector serialization: Serialize, Deserialize, RoundTrip * Tensor 2D serialization: Serialize, Deserialize, RoundTrip * Tensor 3D serialization: Serialize, Deserialize, RoundTrip * JsonConverterRegistry: RegisterCustomConverters * Multiple objects: Serialize and deserialize multiple objects at once * Float vs Double: Performance comparison for different numeric types
- TransferLearningBenchmarks.cs: 16 benchmarks for transfer learning
* Domain Adaptation: CORAL, MMD (with RBF, Linear, Polynomial kernels)
* Feature Mapping: LinearFeatureMapper (Fit, Transform, FitTransform)
* Transfer Algorithms: TransferNeuralNetwork and TransferRandomForest
- Training, fine-tuning, and prediction benchmarks
* End-to-End Scenarios: CORAL+NN, MMD+RF, LinearMapping+NN pipelines
- BENCHMARK_SUMMARY.md: Updated with complete statistics
* Total: 39 files, 607 benchmarks (up from 32 files, 483 benchmarks)
* Coverage: 53+ feature areas (up from 47+)
* New sections added for all 7 new benchmark categories
* Updated performance comparison matrix
* Added "Latest Additions" section documenting all new benchmarks
* Updated benchmark execution examples
New Feature Areas Covered:
31. FitDetectors (20 types) - Overfitting/underfitting detection
32. FitnessCalculators (26+ types) - Model evaluation metrics
33. Interpretability - Fairness, bias detection, explainability
34. OutlierRemoval (5 algorithms) - Data cleaning
35. Caching - ModelCache, GradientCache, key generation
36. Serialization - Matrix, Vector, Tensor JSON performance
37. TransferLearning - Domain adaptation, feature mapping, algorithms
Status: 100% benchmark coverage achieved across all 53+ feature areas
Contributor
|
Warning Rate limit exceeded@ooples has exceeded the limit for the number of commits or files that can be reviewed per hour. Please wait 22 minutes and 11 seconds before requesting another review. ⌛ How to resolve this issue?After the wait time has elapsed, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout. Please see our FAQ for further information. 📒 Files selected for processing (41)
✨ Finishing touches
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Contributor
There was a problem hiding this comment.
Pull Request Overview
This PR adds a comprehensive benchmark suite for the AiDotNet library, providing extensive performance testing infrastructure. The benchmarks cover 53+ feature areas with 607 benchmark methods across 39 files, enabling performance monitoring, regression detection, and competitive analysis against Accord.NET, ML.NET, and TensorFlow.NET.
Key Changes:
- Added BenchmarkSwitcher to Program.cs for flexible benchmark execution with command-line filtering
- Implemented 39 benchmark files covering all major AiDotNet features (LinearAlgebra, Statistics, Regression, Neural Networks, Optimizers, Activation Functions, Loss Functions, etc.)
- Added competitor library dependencies (Accord.NET, ML.NET, TensorFlow.NET) for comparative benchmarking
- Created comprehensive documentation in BENCHMARK_SUMMARY.md detailing all 607 benchmarks
Reviewed Changes
Copilot reviewed 41 out of 41 changed files in this pull request and generated 17 comments.
| File | Description |
|---|---|
| Program.cs | Updated to use BenchmarkSwitcher for command-line benchmark selection |
| AiDotNetBenchmarkTests.csproj | Added competitor library dependencies for benchmarking comparisons |
| BENCHMARK_SUMMARY.md | Comprehensive documentation of all 607 benchmarks across 53+ feature areas |
| VectorOperationsBenchmarks.cs (and 38 other benchmark files) | New benchmark implementations covering all major library features |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+111
to
+117
| foreach (var doc in retrievedDocs) | ||
| { | ||
| if (relevantDocs.Contains(doc)) | ||
| { | ||
| correct++; | ||
| } | ||
| } |
There was a problem hiding this comment.
This foreach loop implicitly filters its target sequence - consider filtering the sequence explicitly using '.Where(...)'.
Comment on lines
+48
to
+55
| if (random.NextDouble() < 0.1) | ||
| { | ||
| _dataWithMissing[i, j] = double.NaN; | ||
| } | ||
| else | ||
| { | ||
| _dataWithMissing[i, j] = value; | ||
| } |
There was a problem hiding this comment.
Both branches of this 'if' statement write to the same variable - consider using '?' to express intent better.
Suggested change
| if (random.NextDouble() < 0.1) | |
| { | |
| _dataWithMissing[i, j] = double.NaN; | |
| } | |
| else | |
| { | |
| _dataWithMissing[i, j] = value; | |
| } | |
| _dataWithMissing[i, j] = (random.NextDouble() < 0.1) ? double.NaN : value; |
Comment on lines
+58
to
+65
| if (random.NextDouble() < 0.05) | ||
| { | ||
| _dataWithOutliers[i, j] = value * 10; // Extreme outlier | ||
| } | ||
| else | ||
| { | ||
| _dataWithOutliers[i, j] = value; | ||
| } |
There was a problem hiding this comment.
Both branches of this 'if' statement write to the same variable - consider using '?' to express intent better.
Suggested change
| if (random.NextDouble() < 0.05) | |
| { | |
| _dataWithOutliers[i, j] = value * 10; // Extreme outlier | |
| } | |
| else | |
| { | |
| _dataWithOutliers[i, j] = value; | |
| } | |
| _dataWithOutliers[i, j] = (random.NextDouble() < 0.05) ? value * 10 : value; // Extreme outlier |
Comment on lines
+43
to
+60
| if (random.NextDouble() < 0.05) // 5% outliers | ||
| { | ||
| _inputs[i, j] = random.NextDouble() * 100; // Large outlier | ||
| } | ||
| else | ||
| { | ||
| _inputs[i, j] = random.NextDouble() * 2 - 1; // Normal data | ||
| } | ||
| } | ||
|
|
||
| if (random.NextDouble() < 0.05) // 5% output outliers | ||
| { | ||
| _outputs[i] = random.NextDouble() * 100; | ||
| } | ||
| else | ||
| { | ||
| _outputs[i] = random.NextDouble() * 10; | ||
| } |
There was a problem hiding this comment.
Both branches of this 'if' statement write to the same variable - consider using '?' to express intent better.
Suggested change
| if (random.NextDouble() < 0.05) // 5% outliers | |
| { | |
| _inputs[i, j] = random.NextDouble() * 100; // Large outlier | |
| } | |
| else | |
| { | |
| _inputs[i, j] = random.NextDouble() * 2 - 1; // Normal data | |
| } | |
| } | |
| if (random.NextDouble() < 0.05) // 5% output outliers | |
| { | |
| _outputs[i] = random.NextDouble() * 100; | |
| } | |
| else | |
| { | |
| _outputs[i] = random.NextDouble() * 10; | |
| } | |
| _inputs[i, j] = (random.NextDouble() < 0.05) | |
| ? random.NextDouble() * 100 // Large outlier | |
| : random.NextDouble() * 2 - 1; // Normal data | |
| } | |
| _outputs[i] = (random.NextDouble() < 0.05) | |
| ? random.NextDouble() * 100 | |
| : random.NextDouble() * 10; |
Comment on lines
+53
to
+60
| if (random.NextDouble() < 0.05) // 5% output outliers | ||
| { | ||
| _outputs[i] = random.NextDouble() * 100; | ||
| } | ||
| else | ||
| { | ||
| _outputs[i] = random.NextDouble() * 10; | ||
| } |
There was a problem hiding this comment.
Both branches of this 'if' statement write to the same variable - consider using '?' to express intent better.
Suggested change
| if (random.NextDouble() < 0.05) // 5% output outliers | |
| { | |
| _outputs[i] = random.NextDouble() * 100; | |
| } | |
| else | |
| { | |
| _outputs[i] = random.NextDouble() * 10; | |
| } | |
| // 5% output outliers | |
| _outputs[i] = (random.NextDouble() < 0.05) | |
| ? random.NextDouble() * 100 | |
| : random.NextDouble() * 10; |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
User Story / Context
merge-dev2-to-masterSummary
Verification
Copilot Review Loop (Outcome-Based)
Record counts before/after your last push:
Files Modified
Notes